A Continuum from Mixtures to Products: Aggregation under Bias

This is a preprint, and does not constitute publication, but is a provided for the benefit of attendees to accompany a talk at the ICML Workshop on Divergence Methods for Probabilistic Inference. If you wish to reference this paper, please reference the final published version. Machine learning models rely heavily on two compositional methods: mixtures and products. Probabilistic aggregation also commonly uses forms of linear opinion pools (which are effectively mixtures), or log opinion pools (which are effectively products). In this paper, we introduce a complete spectrum of compositional methods, Rényi mixtures, that interpolate between mixture models and product models, and hence between log opinion pools and linear opinion pools. We show that these compositional methods are maximum entropy distributions for aggregating information from agents subject to individual biases, with the Rényi divergence parameter dependent on the bias. We also demonstrate practically that Rényi mixtures can provide better performance than log and linear opinion pools, with the optimal limit of log opinion pools when all agents are unbiased and see the same data. We infer that log opinion pools are the most appropriate aggregator for machine learning competitions. We designed, ran and analysed a machine learning Kaggle competition, the results of which confirmed this expectation. Finally we relate Rényi mixtures to recent work on machine learning markets, showing that Rényi aggregators are directly implemented by machine learning markets with isoelastic utilities, and so can result from autonomous self interested decision making by individuals contributing different predictors. Preprint made available for the ICML Workshop on Divergence Methods for Probabilistic Inference, Beijing, China, 2014. Copyright 2014 by the author(s).